Skip to content

Use containerd's content store as Racer's disk copy for Gantry - #1111

Open
Jordan Olshevski (jveski) wants to merge 46 commits into
mainfrom
racer-containerd-store
Open

Jordan Olshevski (jveski) wants to merge 46 commits into
mainfrom
racer-containerd-store

Conversation

@jveski

@jveski Jordan Olshevski (jveski) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This replaces the standalone cachemesh routing library from #1106 by building the same idea into Racer, as suggested in review. A new ClusterCache.spec.persistence: Adapter mode keeps Racer pages in memory only and asks the local origin adapter for a page first (a pinned origin GET with Racer-Source: local), so for Gantry the containerd content store becomes the only on-disk copy; page owners also remember which nodes recently fetched a page and ask those holders before going back to the registry. Racer stays OCI-agnostic and gains a client Invalidate call that drops one cached version across its page owners.

On the Gantry side, Racer mode now checks the caller's credentials against the upstream registry before serving cached content (fixing a gap where any non-empty token could read cached private blobs), and serves local-only page requests from a read-only mount of containerd's content store after verifying each blob's sha256 once. Gantry also verifies the sha256 of manifests and full blob responses it serves from Racer, aborting and invalidating the Racer version on a mismatch. The PR also adds a cluster benchmark harness for comparing default, Racer, and Adapter modes, plus designs/racer-containerd-store.md and updated Racer and Gantry docs.

Racer serves cache hits by digest with no per-request credential check.
Before consulting Racer, HEAD the exact repository and digest upstream with
the caller's delegated Authorization (or none) and cache the decision briefly
per credential hash, registry, repository, kind, and digest.
Racer treats a version's content type as immutable and compares it across
pages. Pages read from the local containerd content store cannot know the
registry's Content-Type header, so registry pages now report blobs as
application/octet-stream too, keeping both sources on one version.
Persistence Adapter keeps a cache's pages in Racer memory only and relies on
the origin adapter's local copies. The CRD rejects it together with
passthrough or origin; the authority maps it to the new CacheMode::Adapter
(wire spelling "adapter"), and publications reject it with a native origin.
A pinned page GET with Racer-Source: local asks the origin adapter to serve
the page only from a copy it holds locally, never from its remote source. A
well-framed 404 or 412 is a local miss. The Go SDK exposes the flag as
OriginRequest.LocalOnly and answers a local-only ErrNotFound or
ErrVersionMismatch with 404.
CacheMode::disk() is false for adapter caches as for passthrough. The store
refuses their writes and pending copies, fills never persist or refresh
them, disk lookups and cuFile staging skip them, reads do not spill ahead to
disk holds, prefetch admits them to memory only, and each snapshot purges
their disk records. Unlike passthrough, pages and metadata stay in memory.
On a memory miss, Fill::acquire_once now visits racer-core's local_sources
before the ranked owner walk. For persistence Adapter that is the node's own
origin adapter, asked with a local-only page request
(Origin::local_page_reserved). A hit is encrypted and published to memory
like an origin page; a miss or failure falls through to peers and the remote
origin without origin failure notes or link-health accounting. Both readers
and owners filling for peers take this path.
DELETE /v1/objects/<key> with a strong If-Match drops the version's metadata,
memory pages, queued writes, and disk index entries on this node, then sends a
signed prefetch 'invalidate' action to each page's ranked owners. Receivers
drop local state only and never forward it.
An invalidation of a version on a worker now also forgets every recorded
holder of each invalidated page, so a later fill does not probe holders
for a version the client declared stale.
…er bench

spec.persistence is mutable, so deploy.sh no longer deletes and recreates
the cache when switching between racer and adapter modes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Empty-metadata probes cannot recover evicted blobs from containerd, and incorrect YAML indentation breaks benchmark deployment with a custom Gantry image.

2 open issues
What changed in this PR

Extends Racer with adapter-backed persistence so Gantry can reuse containerd’s content store instead of keeping a second disk copy, bringing the approach proposed in #1106 into Racer.

Changes:

  • Adds Adapter persistence, local-only page reads, holder probes, and version invalidation.
  • Adds upstream credential checks and full-response digest verification for Gantry’s Racer backend.
  • Adds benchmark tooling, regression tests, and documentation.
File Description
pkg/​racersdk/​origin.go Exposes local-only origin requests and maps local misses.
pkg/​racersdk/​origin_test.go Tests local-only request handling.
pkg/​racersdk/​invalidate.go Adds client-side version invalidation.
pkg/​racersdk/​invalidate_test.go Tests invalidation requests, errors, and cancellation.
pkg/​racersdk/​internal/​wire/​wire.go Encodes and validates local-only headers.
pkg/​racersdk/​internal/​wire/​wire_test.go Tests local-only wire compatibility.
pkg/​racersdk/​internal/​wire/​invalidate.go Implements invalidation request and response encoding.
pkg/​racersdk/​internal/​wire/​invalidate_test.go Tests invalidation protocol validation.
pkg/​racersdk/​doc.go Documents the invalidation API.
internal/​operator/​components/​gantry/​racer.go Mounts containerd’s content store read-only.
internal/​operator/​components/​gantry/​racer_test.go Tests content-store deployment configuration.
internal/​gantry/​racer/​verify.go Verifies streamed digests and invalidates corrupt versions.
internal/​gantry/​racer/​verify_test.go Tests verification, aborted responses, and invalidation.
internal/​gantry/​racer/​racer_test.go Updates blob content-type expectations.
internal/​gantry/​racer/​manifest_test.go Tests normalized blob content types.
internal/​gantry/​racer/​local.go Serves local-only pages from content-addressed storage.
internal/​gantry/​racer/​local_test.go Tests local reads, misses, and metadata consistency.
internal/​gantry/​racer/​handler_test.go Tests aborted full responses on digest mismatch.
internal/​gantry/​racer/​handler_stream_test.go Uses valid digests in stream diagnostics tests.
internal/​gantry/​racer/​access.go Adds cached, credential-scoped upstream access checks.
internal/​gantry/​mirror/​mirror.go Clarifies Racer authorization behavior in comments.
internal/​gantry/​contentstore/​store.go Adds read-only content-store access and digest validation.
internal/​gantry/​contentstore/​store_test.go Tests blob lookup and filesystem safeguards.
internal/​gantry/​contentstore/​open_other.go Provides the non-Linux Unix open implementation.
internal/​gantry/​contentstore/​open_linux.go Adds constrained Linux file opening with fallback.
internal/​gantry/​config/​racer_test.go Tests content-root configuration and validation.
internal/​gantry/​config/​config.go Adds the Racer content-root setting.
hack/​gantry-cluster-bench/​undeploy.sh Restores configuration after benchmarking.
hack/​gantry-cluster-bench/​run.sh Coordinates pulls and collects benchmark results.
hack/​gantry-cluster-bench/​registry.sh Manages the benchmark origin registry.
hack/​gantry-cluster-bench/​prep.sh Configures containerd mirror endpoints.
hack/​gantry-cluster-bench/​manifests/​registry.yaml Defines the benchmark registry deployment.
hack/​gantry-cluster-bench/​manifests/​namespace.yaml Defines the benchmark namespace.
hack/​gantry-cluster-bench/​manifests/​clustercache-racer.yaml Defines the Racer-persistence baseline.
hack/​gantry-cluster-bench/​manifests/​clustercache-adapter.yaml Defines the Adapter-persistence benchmark mode.
hack/​gantry-cluster-bench/​deploy.sh Selects benchmark modes and workload overrides.
hack/​gantry-cluster-bench/​clean.sh Resets target-node content between runs.
hack/​gantry-cluster-bench/​build-gantry-image.sh Builds and pushes benchmark Gantry images.
hack/​cmd/​gantry-cluster-bench/​main.go Adds the benchmark helper command entry point.
hack/​cmd/​gantry-cluster-bench/​config.go Patches Gantry configuration for benchmarks.
hack/​cmd/​gantry-cluster-bench/​config_test.go Tests benchmark configuration patching.
docs/​content/​reference/​racer.md Documents Adapter persistence and invalidation limits.
docs/​content/​guides/​gantry.md Documents setup, authorization, verification, and metrics.
docs/​content/​concepts/​racer.md Explains memory-only Adapter persistence.
cmd/​racer/​wire/​tests/​protocol.rs Tests local-only requests and responses.
cmd/​racer/​wire/​src/​object/​test_util.rs Adds local-copy behavior to origin fixtures.
cmd/​racer/​wire/​src/​object/​origin.rs Implements local-only origin protocol helpers.
cmd/​racer/​wire/​src/​object/​invalidate.rs Defines and tests the invalidation protocol.
cmd/​racer/​wire/​src/​object.rs Exports the invalidation protocol module.
cmd/​racer/​wire/​src/​control.rs Adds Adapter mode and publication validation.
cmd/​racer/​src/​telemetry/​metrics.rs Adds local-read, holder, and invalidation counters.
cmd/​racer/​src/​store/​tests.rs Tests Adapter mode’s disk-write restrictions.
cmd/​racer/​src/​store/​catalog.rs Adds targeted metadata and page invalidation.
cmd/​racer/​src/​store.rs Enforces disk eligibility and discards queued pages.
cmd/​racer/​src/​read/​tests/​peer_copies.rs Tests holder recording, probes, and fallback.
cmd/​racer/​src/​read/​tests/​fill.rs Extends origin fixtures with local-replica responses.
cmd/​racer/​src/​read/​tests/​adapter.rs Tests Adapter reads, fallback, and memory retention.
cmd/​racer/​src/​read/​tests.rs Registers Adapter read tests.
cmd/​racer/​src/​read/​range_stream.rs Disables disk spill for memory-only caches.
cmd/​racer/​src/​read/​metadata.rs Exposes catalog access outside tests.
cmd/​racer/​src/​read/​dispatch.rs Dispatches invalidation and supports memory-only prefetch.
cmd/​racer/​src/​read/​candidates.rs Tracks holders and sends bounded probes.
cmd/​racer/​src/​read.rs Serves holder probes and records readers.
cmd/​racer/​src/​prefetch/​service.rs Handles peer invalidation without forwarding.
cmd/​racer/​src/​peer/​protocol.rs Adds holder and invalidation peer operations.
cmd/​racer/​src/​peer/​forwarding.rs Restricts holder forwarding and tests invalidation messages.
cmd/​racer/​src/​origin/​tests.rs Tests local-only reads over Unix sockets.
cmd/​racer/​src/​origin/​router.rs Routes local reads to adapters only.
cmd/​racer/​src/​origin.rs Implements local-only origin exchanges.
cmd/​racer/​src/​memory.rs Removes cached page copies during invalidation.
cmd/​racer/​src/​control.rs Exposes cache disk eligibility.
cmd/​racer/​src/​client/​tests.rs Tests invalidation through the client socket.
cmd/​racer/​src/​client/​listener.rs Routes client invalidation requests.
cmd/​racer/​src/​client/​invalidate.rs Serves version-specific invalidation requests.
cmd/​racer/​src/​client.rs Registers the invalidation handler.
cmd/​racer/​src/​app.rs Purges Adapter caches’ existing disk records.
cmd/​racer/​README.md Documents persistence, holders, and invalidation internals.
cmd/​racer/​core/​src/​read/​mod.rs Defines local-source selection order.
cmd/​racer/​authority/​src/​rules/​publication.rs Tests publication changes across Adapter transitions.
cmd/​racer/​authority/​src/​rules/​catalog.rs Maps persistence settings to validated cache modes.
cmd/​racer-loadgen/​gantry_test.go Expects mirror aborts for corrupt responses.
cmd/​gantry/​agent_racer.go Wires local storage, access checks, and invalidation.
cmd/​gantry/​agent_racer_test.go Tests origin options and updates client fixtures.
api/​racer/​v1alpha1/​crd/​racer.unbounded-cloud.io_clustercaches.yaml Adds persistence schema and compatibility constraints.
api/​racer/​v1alpha1/​clustercache_types.go Defines persistence values and effective defaults.
api/​racer/​v1alpha1/​clustercache_test.go Tests persistence defaults and serialization.
api/​racer/​v1alpha1/​clustercache_envtest_test.go Tests API validation and persistence transitions.

Review effort: Balanced

Comment thread cmd/racer/src/read.rs Resolved
Comment thread hack/gantry-cluster-bench/deploy.sh Resolved
…fixture reads

The retained-identity startup test gave bootstrap a 30s scope, but seat
403 rejections retry 5 times with ~1+2+4+8+16s backoff, so loaded runs
hit DeadlineExceeded before the retry bound reported Unauthorized. Use a
120s scope. Also raise the portable adapter fixture client socket
timeouts from 2s to 30s.
A holder whose memory copy was evicted asked its local adapter with no
metadata, so an adapter that names objects by metadata (Gantry) could
never serve its durable copy. Owners now forward the reader's opaque
adapter metadata, never authorization, in holder probes, and holders
pass it to their local-only adapter request.
Comment thread internal/gantry/racer/verify.go Fixed
Comment thread internal/gantry/racer/verify.go Fixed
Without metadata the key and ETag cannot distinguish a manifest from a
blob, so no Content-Type is guaranteed to match the registry's. Answer
ErrNotFound so Racer moves on instead of failing the read.
The verifying writer only forwards registry object bytes, which are already served as attachments with nosniff. Holding the destination as an io.Writer keeps CodeQL from treating the passthrough as an HTML response sink, and the stale comment about digest verification now matches the full-object check.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local replica publication errors and stalls can prevent fallback to healthy sources, while inconsistent manifest Content-Type metadata can cause valid pulls to fail.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Unbounded probe timeout blocks peer and origin fallback

cmd/​racer/​src/​read/​fill.rs:2967

A local adapter that accepts the socket but stalls can consume the entire read deadline because this optional probe uses the parent scope for both headers and body. Its DeadlineExceeded is propagated at line 2993, so healthy peers or the remote origin are never tried. Give the probe a bounded child scope and treat that child's timeout as a local failure while the parent remains live.

Medium severity Publication errors abort peer and origin fallback

cmd/​racer/​src/​read/​fill.rs:2985

A local replica whose immutable metadata conflicts with the retained version reaches publish, which returns CorruptRecord; this ? then fails the whole read instead of trying peers or the remote origin. Move publication into attempt so that publication failures use the existing local-failure fallback.

Medium severity Manifest media type mismatch breaks immutable metadata

internal/​gantry/​racer/​local.go:171

A manifest's JSON mediaType can omit parameters from the registry's Content-Type, such as charset=utf-8, while originContentType preserves that header (racer.go:295-300). Returning the unparameterized value under the same ETag conflicts with Racer's immutable metadata checks and can fail a valid pull. Preserve the authoritative manifest type across both sources, or return a local miss when it cannot be reproduced.

2 resolved since last review

Review effort: Balanced

Reduce manifest content types to the bare lowercase media type on both the registry and local content store paths so Racer version metadata agrees for the same ETag.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved issues break custom-namespace setup, allow corrupt adapter replicas to cause repeated verification failures, and reuse incorrect media types across blob and manifest routes.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Render BENCH_NS consistently in both manifests

hack/​gantry-cluster-bench/​registry.sh:35

BENCH_NS is configurable, but both applied manifests hard-code gantry-bench, and this rendering only substitutes the node and port. Setting another namespace therefore deploys the registry in gantry-bench while the rollout wait and all subsequent registry commands target BENCH_NS, so setup fails. Render the configured namespace into both manifests (and the namespace creation used by the DaemonSet helpers).

Medium severity Preserve manifest media type across digest cache hits

internal/​gantry/​racer/​local.go:152

Request keys objects only by digest (racer.go:88), but this descriptor changes with the request kind. After a blob-route read caches a manifest as application/octet-stream, a manifest-route read of the same digest reuses that type (racer.go:858) instead of its manifest/index media type, which can break image resolution. Make cached metadata route-independent and recover the manifest media type at the HTTP boundary, or separate cache identities; add a blob-then-manifest regression test.

Medium severity Bypass corrupt adapter replicas during recovery reads

internal/​gantry/​racer/​verify.go:152

When a multi-page blob is corrupt in an adapter's containerd store, Invalidate clears Racer state but serveLocal can reopen the same bad bytes on the next read (internal/gantry/racer/local.go:111). This can make retries repeatedly fail digest verification even if registry content is healthy. Add a quarantine or bypass mechanism for suspect adapter replicas, including those on remote owners, so recovery reads can reach the registry.

Review effort: Balanced

Hash each content store file once, keyed by file identity, before serving
it for a local-only request. Corrupt files are quarantined as local misses
until replaced, so recovery after Invalidate can reach peers or the
registry instead of rereading bad bytes on this node or a remote owner.
A mirror digest mismatch drops the local result so the file is rechecked.
Adds gantry_racer_local_replica_verifications_total{result}.
Racer keys objects by digest alone, so a digest filled through the blob
route was later served on the manifest route as application/octet-stream.
Store application/octet-stream for every kind from both the registry and
local content store, and recover the manifest media type at the HTTP
boundary from the buffered, digest-checked manifest body (mediaType field,
else inferred OCI index or manifest; untypeable bodies get 503 so
containerd falls through to the registry).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The fallback open omits O_NONBLOCK, allowing a FIFO at a digest path to block indefinitely before validation and occupy an origin callback slot despite the request deadline.

1 open issue

Review effort: Balanced

Comment thread internal/gantry/contentstore/store.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The coupled Go/Rust persistence, peer-recovery, and credential-enforcement changes require human review and multi-node validation of mode transitions and corruption recovery under concurrent pulls and peer failures.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@jveski
Jordan Olshevski (jveski) marked this pull request as ready for review November 11, 2026 01:39
@jveski
Jordan Olshevski (jveski) requested a review from a team November 11, 2026 01:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants